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GOVERNMENT RIGHTS NOTICE 
Portions of the material in this specification arose as a result of Government 
support under grants MH58964 and MH58964-02 between Clever Sys., Inc. and The 
National Institute of Mental Health, National Institute of Health. The Government 
1 0 has certain rights in this invention. 

BACKGROUND OF THE INVENTION 

1. Technical Field 

15 The invention relates generally to behavior analysis of animal objects. More 

particularly, one aspect of the invention is directed to monitoring and characterization 
of behaviors under specific behavioral paradigm experiments, including home cage 
behavior paradigms, locomotion or open field paradigm experiment, object 
recognition paradigm experiments, variety of maze paradigm experiments, water 

20 maze paradigm experiments, freezing paradigm experiments for conditioned fear, for 
an animal, for example, a mouse or a rat, using video analysis from a top view image 
or side view image, or the integration of both views. 

2. Background Art 
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Animals, for example mice or rats, are used extensively as human models in 
the research of drug development; genetic functions; toxicology research; 
understanding and treatment of diseases; and other research applications. Despite the 
differing lifestyles of humans and animals, for example mice, their extensive genetic 
5 and neuroanatomical homologies give rise to a wide variety of behavioral processes 
that are widely conserved between species. Exploration of these shared brain 
functions will shed light on fundamental elements of human behavioral regulation. 
Therefore, many behavioral test experiments have been designed on animals like mice 
and rats to explore their behaviors. These experiments include, but not limited to, 

10 home cage behaviors, open field locomotion experiments, object recognition 
experiments, a variety of maze experiments, water maze experiments, and freezing 
experiments for conditioned fear. 

Animal's home cage activity patterns are important examination item on the 
general health list of animals, such as mice and rats. It provides many important 

15 indications of whether the animal's health status is normal or abnormal. Home cage 
behaviors are best observed by videotaping several 24-hour periods in the animal 
housing facility, and subsequent scoring of the videotape by two independent 
observers. However, this observation has rarely been done until our inventions came 
into play, due to the instability in long term human observation, the time consumed, 

20 and the huge costs associated with the observation. 

A conventional method for measuring animal's spatial navigation learning and 
memory is the water maze task, in which the animal swims to find a hidden platform, 
using visual cues to locate the platform. This task is based on the principle that 

2 
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rodents are highly motivated to escape from a water environment by the quickest, 
most direct route. (Wenk, 1997) Experiment sessions are usually videotaped, and 
human observation of videotapes or automated software is used, depending on the 
parameters required for observing the session. 
5 Variations of popular methods for measuring animal's spatial navigation 

learning and memory include other maze designs, such as T-maze, Y-maze, and radial 
arm maze. In all cases, the task requires the animal to choose specific arm(s) of the 
maze to receive a food or water reinforcement or to avoid a footshock. The shapes of 
the arms make the differences among T-maze, Y-maze, and radial arm maze. The 

10 animal is habituated and then shaped to obtain the reinforcer. Variations of popular 
methods for measuring animal's anxiety-related behaviors include elevated plus maze, 
zero maze, etc. Measuring anxiety using elevated plus maze or zero maze rests on the 
naturalistic conflict between the tendency of animal such as mice to explore a novel 
environment and the aversive properties of a brightly lit, open area. Elevated plus 

15 maze is elevated from the ground about a meter with four (4) arms, two well lit and 
two closed and dark. Animals such as mice or rats prefer the closed arms but will 
venture out into the open arms, with a start box in the center. The zero maze is 
similar, but has annulus of an elevated circular runway, with areas brightly lit 
alternate with dark, covered areas. We group all of these experiments under "maze". 

20 Though the theory and operations and training may be different among these mazes, 
the observation and measurement is basically similar, i.e., the measurement of animal 
staying in each arm or arena, closed or open. Experiment session is usually 
videotaped, and human observation of videotapes or automates software is used, 
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depending on the parameters required for observing the session. 

Another method to measure animal's such as mice or rats, capability in spatial 
learning and memory and their tendency of exploration is the experiment of object 
recognition, or novelty seeking. Its objective is to measure reduced time spent 
5 exploring a novel object that replaced a training object after a specified retention time. 
Objects of different shapes and colors are placed in an open field, and animal is place 
in the field. The number of times the animal sniffs at each object, and the duration of 
each sniffing are measured to show the animal's tendency to explore. Objects are 
replaced with new objects from time to time. The experiment session is videotaped, 

10 and human observation of videotapes is used to measure those parameters. 

The most standardized general measure of motor function is spontaneous 
activity in the open field. Square, rectangular, and circular equipment is presently in 
common use. Sizes of open fields range from centimeters to several meters. Scoring 
of videotaped session allows quantization of animal's spontaneous activity. 

15 Automated open fields now routinely used in behavioral neuroscience laboratories are 
equipped with either photocell beams or video tracking and computer software. Both 
types of automated systems calculate a useful range of basic locomotor parameters. 

Freezing test is designed for cued and contextual fear conditioning, which is 
among the most intuitive memory paradigms. Freezing is a common response feared 

20 situation in many species, and is defined as no movements other than respiration. 
Conditioning training consists of placing the mouse in the chamber and exposing to a 
mild footshock paired with auditory cue. Freezing is measured when the trained 
mouse is placed back in the same chamber for training with auditory cue, and scored 
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for bouts of freezing behavior. Human observation of videotapes of the session is 
used for this scoring, which is inaccurate and expensive. Automated system using 
mechanical principle exists to help real-time scoring of freezing. However, the 
precision of such a mechanical system need to be further improved. 
5 As discussed, all these apparatus and experiments use, in many cases, human 

observation of videotapes of the experiment sessions, resulting in inaccuracy, 
subjectivity, labor-intensive, and thus expensive experiments. Some automating 
software provides rudimentary and basic parameters, relying on tracking animal as a 
point in space, generating experiment results that are inaccurate and can not meet the 

10 demands for advanced features. Besides, each system software module works for 
only a specific experiment, resulting in potential discrepancy in the results across 
different systems due to differences in software algorithms used. 

All the observations of these behavioral experiments use video to record 
experiment processes and rely human observations. This introduces the opportunity 

15 to utilize the latest technologies development in computer vision, image processing, 
and digital video processing to automate the processes and achieve better results, high 
throughput screening, and lower costs. Many of these experiments are conducted 
with observations performed from top view, that is, observation of the experiments 
from above the apparatus is used to obtain needed parameters. This also provides an 

20 opportunity to unify the approaches to observe and analyze these experiments' results. 



SUMMARY OF THE INVENTION 
There are strong needs for automated systems and software that can automate 
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the measurements of the experiments mentioned above, provide the measurements of 
meaningful complex behaviors and revealing new parameters that characterize animal 
behaviors to meet post-genomic era's demands, and obtain consistent results using 
novel approaches. 

5 A revolutionary approach is invented to automatically measure animal's home 

cage activity patterns. This approach consists of defining a unique set of animal's, 
such as mice or rats, behavior category. This category includes behaviors like rearing, 
walking, grooming, eating, drinking, jumping, hanging, etc. Computer systems are 
designed and implemented that can product digital video files of animal's behaviors in 

10 a home cage in real time or off-line mode. Software algorithms are developed to 
automatically understand the animal's behaviors in those video files. 

A novel and unified framework is provided for automatically analyzing animal 
behaviors from multiple different behavioral paradigms. This unified framework lays 
the foundation and constitutes the common layer for all the automated measurements 

15 in the experiments of water maze, maze, locomotion, object recognition, and freezing 
for fear conditioning. 

Creative algorithms are designed to support innovative analysis of behaviors 
of animals such as mice or rats. This analysis is based on the premise that the entire 
animal body, body parts, related color information, and their dynamic motion are 

20 taken advantage of in order to provide the measurement of complex behaviors and 
novel parameters. 

Virtual apparatus is designed and implemented in the system to ensure same 
software framework can be applied to different apparatuses such as water maze, 
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variations of mazes including T-maze, Y-maze, radial arm maze, elevated plus maze, 
zero maze, cages of rectangular, circular, or any other shapes, and object of any color 
and shapes, instead of having different software component to handle different 
apparatus,. . The software is designed to provide graphic tools, and users can use 
5 these graphic tools to create virtual apparatus corresponding to the real apparatus that 
is being used for the experiment under observation. Graphic tools also provide the 
capability to calibrate these apparatus, allowing better identification of behaviors and 
precise measurement of behaviors in real measurements instead of relative 
measurement using number of pixels. 

10 Virtual zones are another invention that has been implemented in the system. 

In all the experiments, the animal is moving around the cage, and the activity 
distribution in different zones of the cage is of great interest to many scientists. 
Convention approach is to use infrared photobeams to divide the cage into zones 
needed. Photobeams are released from tubes at one end, and sensor on the opposite 

15 end receives the photobeams. When photobeams are interrupted by animal, the 
receiver records the signal. In this way, the animal can move around while their 
activity distribution across zones is recorded and calculated. In this invention, virtual 
zones are used in stead of the zones created by photobeams. Graphic tools are 
designed and provided for the users. Users draw the zones as they want using the 

20 graphic tools. Software tracks the animals and records how the animal crosses the 
zones and stay in the zones as defined by the drawings. 

Another invention is that algorithms are designed and implemented to allow 
different modules to be combined to achieve multiple experiment purposes in a joint 
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operation. For one example, measuring animal's object recognition behaviors and 
measuring animal's locomotion activity can be performed during one experiment 
operation. The users can perform their regular object recognition experiment. 
However, software modules of object recognition and locomotion analysis are both 
5 executed to obtain analysis results of both. This flexibility enhances the power and 
usability of the system. 

In general, the present invention is directed to systems and methods for 
finding patterns of behaviors and/or activities of an animal using video. The 
invention includes a system with a video camera connected to a computer in which 

10 the computer is configured to automatically provide animal identification, animal 
motion tracking (for moving animal), animal shape, animal body parts, and posture 
classification, and behavior identification. Thus, the present invention is capable of 
automatically monitoring a video image to identify, track and classify the actions of 
various animals and their movements. The video image may be provided in real time 

15 from a camera and/or from a storage location. The invention is particularly useful for 
monitoring and classifying mice or rats behavior for testing drugs and genetic 
mutations, but may be used in any of a number of surveillance or other applications. 

In one embodiment the invention includes a system in which an analog/digital 
video camera and a video record/playback device (e.g., VCR) are coupled to a video 

20 digitization/compression unit. The video camera may provide a video image 
containing an animal to be identified. The video digitization/compression unit is 
coupled to a computer that is configured to automatically monitor the video image to 
identify, track and classify the actions of the animal and its movements over time 
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within a sequence of video session image frames. The digitization/compression unit 
may convert analog video and audio into, for example, MPEG or other formats. The 
computer may be, for example, a personal computer, using either a Windows platform 
or a Unix platform, or a Macintosh computer and compatible platform. The computer 
5 is loaded and configured with custom software programs (or equipped with firmware) 
using, for example, MATLAB or C/C++ programming language, so as to analyze the 
digitized video for animal identification and segmentation, tracking, and/or 
behavior/activity characterization. This software may be stored in, for example, a 
program memory, which may include ROM, RAM, CD ROM and/or a hard drive, etc. 

10 In one variation of the invention the software (or firmware) includes a unique 
background subtraction method which is more simple, efficient, and accurate than 
those previously known. 

In operation, the system receives incoming video images from either the video 
camera in real time or pre-recorded from the video record/playback unit. If the video 

15 is in analog format, then the information is converted from analog to digital format 
and may be compressed by the video digitization/compression unit. The digital video 
images are then provided to the computer where various processes are undertaken to 
identify and segment a predetermined animal from the image. In a preferred 
embodiment the animal is a mouse or rat in motion with some movement from frame 

20 to frame in the video, and is in the foreground of the video images. In any case, the 
digital images may be processed to identify and segregate a desired (predetermined) 
animal from the various frames of incoming video. This process may be achieved 
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using, for example, background subtraction, mixture modeling, robust estimation, 
and/or other processes. 

The shape and location of the desired animal is then tracked from one frame or 
scene to another frame or scene of video images. The body parts of the animal such 
5 as head, mouth, and tail are identified by novel approaches through body contour 
segmentation, contour segment classification, and relaxation labeling. Next, the 
changes in the shapes, locations, body parts, and/or postures of the animal of interest 
may be identified, their features extracted. Then, the shape, location, body parts, and 
other related information may be used to characterize the animal's activity into one of 

1 0 a number of pre-defined behaviors. For example, if the animal is a mouse or rat, 
some pre-defined normal behaviors may include sleeping, walking, sniffing, etc., and 
pre-defined abnormal behavior may include spinning vertical, jumping in the same 
spot, etc. The pre-defined behaviors may be stored in a database in the data memory. 
The behavior may be characterized using, for example, approaches such as rule-based 

15 analysis, token parsing procedure, and/or Hidden Markov Modeling (HMM). Further, 
the system may be constructed to characterize the object behavior as new behavior 
and particular temporal rhythm. 

In another preferred embodiment directed towards the video camera providing 
a video image containing animals such as mice or rats to be identified, the system 

20 operates as follows. There is at least one camera, or multiple cameras, taking video 
image of experiment apparatus that contain animals. There is at least one apparatus, 
or as many as the computer computing power allows, say four (4) or sixteen (16) or 
even more. Each apparatus contains at least one animal or multiple animals. The 
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multiple cameras may be taking video from different points of views such as one 
taking video images from the side of the apparatus, or one taking video images from 
the top of the apparatus. These apparatus can be home cage, open field cage, water 
maze device, T-maze device, Y-maze device, radial arm device, zero maze device, 
5 elevated plus maze device, or other experiment devices. When video images are 
taken of multiple apparatuses and devices containing one or multiple animals, and are 
analyzed for identifying these animals' behaviors, high throughput screening is 
achieved. When video images taken from different points of views, for example, one 
from the top view and another from the side view, are combined to identify animal's 

1 0 behaviors, integrated analysis is achieved. 

In another preferred embodiment directed toward video analysis of animals 
such as mice or rats, the system operates as follows. As a preliminary matter, normal 
postures and behaviors of the animals are defined and may be entered into a Normal 
Paradigm Parameters, Postures and Behaviors database. In analyzing, in a first 

15 instant, incoming video images are received. The system determines if the video 
images are in analog or digital format and input into a computer. If the video images 
are in analog format they are digitized and may be compressed, using, for example, an 
MPEG digitizer/compression unit. Otherwise, the digital video image may be input 
directly to the computer. Next, a background may be generated or updated from the 

20 digital video images and foreground objects detected. Next, the foreground animal 
features are extracted. Also, body parts such as head, tail, ear, mouth, forelimbs, hind 
limbs, abdomen, and upper and lower back, are identified. Two different methods are 
pursuing from this point, depending on different behavior paradigms. In one method, 
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the foreground animal shape is classified into various categories, for example, 
standing, sitting, etc. Next, the foreground animal posture is compared to the various 
predefined postures stored in the database, and then identified as a particular posture 
or a new (unidentified) posture. Then, various groups of postures and body parts are 
5 concatenated into a series to make up a foreground animal behavior compared against 
the sequence of postures, stored in for example a database in memory, that make up 
known normal or abnormal behaviors of the animal. The abnormal behaviors are then 
identified in terms of known abnormal behavior, new behavior and/or daily rhythm. 
In another method, behavioral processes and events are detected, and behavior 
10 parameters are calculated. These behaviors parameters give indications to animal 
health information related to learning and memory capability, anxiety, and relations to 
certain diseases. 

In one variation of the invention, animal detection is performed through a 
unique method of background subtraction. First, the incoming digital video signal is 

15 split into individual images (frames) in real-time. Then, the system determines if the 
background image derived from prior incoming video needs to be updated due to 
changes in the background image or a background image needs to be developed 
because there was no background image was previously developed. If the 
background image needs to be generated, then a number of frames of video image, for 

20 example 20, will be grouped into a sample of images. Then, the system creates a 
standard deviation map of the sample of images. Next, the process removes a 
bounding box area in each frame or image where the variation within the group of 
images is above a predetermined threshold (i.e., where the object of interest or 
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moving objects are located). Then, the various images within the sample less the 
bounding box area are averaged. Final background is obtained by averaging 5-10 
samples. This completes the background generation process. However, often the 
background image does not remain constant for a great length of time due to various 
5 reasons. Thus, the background needs to be dynamically recalculated periodically as 
above or it can be recalculated by keeping track of the difference image and note any 
sudden changes. The newly dynamically generated background image is next 
subtracted from the current video image(s) to obtain foreground areas that may 
include the object of interest. 

10 Next, the object identification/detection process is performed. First, regions of 

interest (ROI) are obtained by identifying areas where the intensity difference 
generated from the subtraction is greater than a predetermined threshold, which 
constitute potential foreground object(s) being sought. Classification of these 
foreground regions of interest will be performed using the sizes of the ROIs, distances 

15 among these ROIs, threshold of intensity, and connectedness, to thereby identify the 
foreground objects. Next, the foreground object identification/detection process may 
be refined by adaptively learning histograms of foreground ROIs and using edge 
detection to more accurately identify the desired object(s). Finally, the information 
identifying the desired foreground object is output. The process may then continue 

20 with the tracking and/or behavior characterization step(s). 

The previous embodiments are particularly applicable to the study and 
analysis of mice or rats used in genetic and drug experimentation. Another variation 
of the present invention is directed to automatically determining locomotion behavior 
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of mice or rats in an open field. Once the animal is identified as foreground object as 
discussed above, the body parts of the animal such as head and tail, and hind limbs 
and forelimbs, and center of mass, are identified. The traces of the path of the 
movements of the animal's center of mass in the open field under observation is 
5 recorded, its instant and average speed of movements and distance traveled are 
calculated, its instant and cumulative body turning angles are analyzed. In addition, 
events like turning ratio (ratio of path length over number of turns, where number of 
turns is counted when the animal makes a turn larger than 80 degrees when the animal 
travels one body length); proximity score (calculated by determining the distance of 

10 the animal from the goal during each second of the trial and is used as a measure of 
deviation from the ideal path to the platform once an animal is placed in the cage); 
heading errors (defined as an instance of swimming away from the VISIBLE 
platform); and animal staying in a specific zone inside the field, are recorded. Then a 
visualization process will further analyze the result of the path trace recorded to 

15 generate variety of statistic results. Visualization process allows users to use graphic 
drawing tools to define any number of zones of any shape in the open field as needed. 
The system provides a graphic tool that allows users to define the field of any shape. 

The previous embodiments are particularly applicable to the study and 
analysis of mice or rats in their capability to explore new objects. Another variation 

20 of the present invention is directed to automatically determining the object 
recognition activity. Graphic tools are provided to allow users to define objects in the 
scene. Once the animal is identified as a foreground object as discussed above, the 
body parts of the animal such as head and tail and hind limbs and forelimbs, and 



PATENT APPLICATION 
Attorney Docket No.258/160 

center of mass are identified. The traces of the path of the movements of the animal's 
center of mass are recorded. The distances of the animal's head to any of the objects 
in the scene are calculated and when the distance to an object is less than a user- 
defined amount, the animal is counted as animal's sniffing at the object and is said to 
5 be exploring that object. Statistics about these exploring events are generated and 
exported. 

The previous embodiments are particularly applicable to the study and 
analysis of mice or rats in their spatial learning and memory. Third variation of the 
present invention is directed to automatically determining the behaviors of mice or 

10 rats in a water maze experiment environment. Graphic tools are provided to allow 
users to define the maze and platforms. Once the animal is identified as a foreground 
object as discussed above, the body parts of the animal such as head and tail and hind 
limbs and forelimbs, and center of mass are identified. The traces of the path of the 
movements of the animal's center of mass are recorded. The latency (the time period 

15 the animal spent in swimming in the water before landing at the platform) is 
measured; its instant and average speed of movements and distance traveled are 
calculated; its instant and cumulative body turning angles are analyzed. In addition, 
events like turning ratio (ratio of path length over number of turns, where number of 
turns is counted when the animal makes a turn larger than 90 degrees when the animal 

20 travels one body length); proximity score (calculated by determining the distance of 
the animal from the goal (platform) during each second of the trial and is used as a 
measure of deviation from the ideal path to the platform once an animal is placed in 
the water); heading errors (defined as an instance of swimming away from the 
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VISIBLE platform); and animal staying in a specific zone inside the maze, are 
recorded. Then a visualization process will further analyze the result of the path trace 
recorded to generate variety of statistic results. Visualization process allows users to 
use graphic drawing tools to define any number of zones of any shape in the open 
5 field as needed. The system provides a graphic tool that allows users to define the 
field of any shape. 

The previous embodiments are particularly applicable to the study and 
analysis of mice or rats in their spatial learning and memory and anxiety. Fourth 
variation of the present invention is directed to automatically determining the 

10 behaviors of mice or rats in a variety of maze apparatus. Graphic tools are provided 
to define specific maze apparatus, such as T-maze, Y-maze, radial arm maze, zero 
maze, elevated plus maze, and etc. Once the animal is identified as a foreground 
object as discussed above, the body parts of the animal such as head and tail and hind 
limbs and forelimbs, and center of mass are identified. The traces of the path of the 

15 movements of the animal's center of mass are recorded. More importantly, the 
animal's behaviors related to every arm in the maze, such as time spent in each arm, 
the number of times entering and exiting an arm, are found. Besides, anima's instant 
and average speed of movements and distance traveled are calculated; its instant and 
cumulative body turning angles are analyzed. In addition, events such as animal 

20 partial incursions into particular arm (for example, the animal might maintain its hind 
quarters in a closed arm while poking its nose into an open arm); Stretch-Attend 
Behavior; Head-Dipping behavior; and Supported Rearing, are detected. 
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Fifth variation of the present invention is directed to automatically 
determining the freezing behaviors of mice or rats in a cued or conditioned fear tests. 
Graphic tools are provided. Graphic tools are provided to define the area within 
which animal activity is measured. Differences between neighboring frames are 
5 compared pixel-by-pixel in terms of their intensity and color intensity. These 
differences are used to calculate the motion of the animal from frame-to-frame 
because motion in the area is caused by movements of the animal. The values of 
these differences indicate if the animal is moving or freezing. 

Development activities have been completed to validate various scientific 

10 definitions of mouse behaviors and to create novel digital video processing algorithms 
for mouse tracking and behavior recognition, which are embodied in a software and 
hardware system according to the present invention. An automated method for 
analysis of mouse behavior from digitized 24 hours video has been achieved using the 
present invention and its digital video analysis method for object identification and 

15 segmentation, tracking, and classification. Several different methods and their 
algorithms, including Background Subtraction, Probabilistic approach with 
Expectation-Maximization, and Robust Estimation to find parameter values by best 
fitting a set of data measurements and results proved successful. 

The previous embodiments are particularly applicable to the study and 

20 analysis of mice or rats used in genetic and drug experimentation. One variation of 
the present invention is directed particularly to automatically determining the 
behavioral characteristics of a mouse in a home cage, a cage looking like a shoebox 
used for housing animals. The need for sensitive detection of novel phenotypes of 
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genetically manipulated or drug-administered mice demands automation of analyses. 
Behavioral phenotypes are often best detected when mice are unconstrained by 
experimenter manipulation. Thus, automation of analysis of behavior in a known 
environment, for example a home cage, would be a powerful tool for detecting 
5 phenotypes resulting from gene manipulations or drug administrations. Automation 
of analysis would allow quantification of all behaviors as they vary across the daily 
cycle of activity. Because gene defects causing developmental disorders in humans 
usually result in changes in the daily rhythm of behavior, analysis of organized 
patterns of behavior across the day may also be effective in detecting phenotypes in 

10 transgenic and targeted mutant mice. The automated system may also be able to 
detect behaviors that do not normally occur and present the investigator with video 
clips of such behavior without the investigator having to view an entire day or long 
period of mouse activity to manually identify the desired behavior. 

The systematically developed definition of mouse behavior that is detectable 

15 by the automated analysis according to the present invention makes precise and 
quantitative analysis of the entire mouse behavior repertoire possible for the first time. 
The various computer algorithms included in the invention for automating behavior 
analysis based on the behavior definitions ensure accurate and efficient identification 
of mouse behaviors. In addition, the digital video analysis techniques of the present 

20 invention improves analysis of behavior by leading to: (1) decreased variance due to 
non-disturbed observation of the animal; (2) increased experiment sensitivity due to 
the greater number of behaviors sampled over a much longer time span than ever 
before possible; and (3) the potential to be applied to all common normative behavior 
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patterns, capability to assess subtle behavioral states, and detection of changes of 
behavior patterns in addition to individual behaviors. 

The invention may identify some abnormal behavior by using video image 
information (for example, stored in memory) of known abnormal animals to build a 
5 video profile for that behavior. For example, video image of vertical spinning while 
hanging from the cage top was stored to memory and used to automatically identify 
such activity in mice. Further, abnormalities may also result from an increase in any 
particular type of normal behavior. Detection of such new abnormal behaviors may 
be achieved by the present invention detecting, for example, segments of behavior 
10 that do not fit the standard profile. The standard profile may be developed for a 
particular strain of mouse whereas detection of abnormal amounts of a normal 
behavior can be detected by comparison to the statistical properties of the standard 
profile. 

Thus, the automated analysis of the present invention may be used to build 
15 profiles of the behaviors, their amount, duration, and daily cycle for each animal, for 
example each commonly used strain of mice. A plurality of such profiles may be 
stored in, for example, a database in a data memory of the computer. One or more of 
these profiles may then be compared to a mouse in question and difference from the 
profile expressed quantitatively. 
20 The techniques developed with the present invention for automation of the 

categorization and quantification of all home-cage mouse behaviors throughout the 
daily cycle is a powerful tool for detecting phenotypic effects of gene manipulations 
in mice. As previously discussed, this technology is extendable to other behavior 
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studies of animals and humans, as well as surveillance purposes. As will be described 
in detail below, the present invention provides automated systems and methods for 
automated accurate identification, tracking and behavior categorization of an object 
whose image is captured with video. 
5 Other variations of the present invention is directed particularly to 

automatically determining the behavioral characteristics of an animal in various 
behavioral experiment apparatus such as water maze, Y-maze, T-maze, zero maze, 
elevated plus maze, locomotion open field, field for object recognition study, and 
cued or conditioned fear. In these experiment apparatuses, animal's body contour, 

10 center of mass, body parts including head, tail, forelimbs, hind limbs and etc. are 
accurately identified using the embodiments above. This allows excellent 
understanding of animal's behaviors within these specific experiment apparatus and 
procedures. Many novel and important parameters, which were beyond reach 
previously, are now successfully analyzed. These parameters include, but not limited 

15 to, traces of path of animal's center of mass, instant and average speed, instant and 
average of body turning angles, distance traveled, turning ratio, proximity score, 
heading error, stretch-and-attend, head-dipping, stay-across-arms, supported-rearing, 
sniffing (exploring) at particular objects, latency time to get to the goal (platform), 
time spent in specific arm/arena or specific zones within arm/arena, number of time 

20 entering and exiting arm/arena or specific zones within arm/arena, and etc. These 
parameters provide good indications for gene targeting, drug screening, toxicology 
research, learning and memory process study, anxiety study, understanding and 
treatment of diseases such as Parkinson's Disease, Alzheimer Disease, ALS, and etc. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a block diagram of one exemplary system configurable to find the 
position, shape, and behavioral characteristics of an object using automated video 
5 analysis, according to one embodiment of the present invention. 

Figure 2 is a flow chart of a method of automatic video analysis for object 
identification and characterization, according to one embodiment of the present 
invention. 

Figure 3 is a flow chart of a method of automatic video analysis for animal 
10 identification and characterization from video shot from the top, according to another 
embodiment of the present invention. 

Figure 4 shows an embodiment of the invention where video is captured from 

the top. 

Figure 5 shows another embodiment of the invention, a top-view based open 
1 5 field locomotion analysis system. Multiple arenas can be analyzed at the same time. 

Figure 6 shows another embodiment of the invention, a top-view based maze 
behavior analysis system. An elevated plus maze is shown, but other types of mazes 
such as zero maze, T-maze, Y-maze, radial arm maze can be used. 

Figure 7 shows another embodiment of the invention, a top-view based object 
20 recognition behavior analysis system. A plurality of objects is placed in the arena and 
sniffing behavior of the animal on the objects is detected. Multiple arenas can be 
analyzed at the same time. 
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Figure 8 shows another embodiment of the invention, a top-view based water 
maze behavior analysis system. The platform is the target for the animal. 

Figure 9 shows another embodiment of the invention, a freezing behavior 
analysis system. The animal is placed inside a chamber where stimuli such as electric 
5 shock or auditory tones are used and the resulting behavior after the stimuli are 
observed. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The past few years have seen an increase in the integration of video camera 

10 and computer technologies. Today, the integration of the two technologies allows 
video images to be digitized, stored, and viewed on small inexpensive computers, for 
example, a personal computer. Further, the processing and storage capabilities of 
these small inexpensive computers has expanded rapidly and reduced the cost for 
performing data and computational intensive applications. Thus, video analysis 

15 systems may now be configured to provide robust surveillance systems that can 
provide automated analysis and identification of various objects and characterization 
of their behavior. The present invention provides such systems and related methods. 

In general, the present invention can automatically find the patterns of 
behaviors and/or activities of a predetermined object being monitored using video. 

20 The invention includes a system with a video camera connected to a computer in 
which the computer is configured to automatically provide object identification, 
object motion tracking (for moving objects), object shape classification, and behavior 
identification. In a preferred embodiment the system includes various video analysis 
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algorithms. The computer processes analyze digitized video with the various 
algorithms so as to automatically monitor a video image to identify, track and classify 
the actions of one or more predetermined objects and its movements captured by the 
video image as it occurs from one video frame or scene to another. The system may 
5 characterize behavior by accessing a database of object information of known 
behavior of the predetermined object. The image to be analyzed may be provided in 
real time from one or more camera and/or from storage. 

In various exemplary embodiments described in detail as follows, the 
invention is configured to enable monitoring and classifying of animal behavior that 

10 result from testing drugs and genetic mutations on animals. However, as indicated 
above, the system may be similarly configured for use in any of a number of 
surveillance or other applications. For example, the invention can be applied to 
various situations in which tracking moving objects is needed. One such situation is 
security surveillance in public areas like airports, military bases, or home security 

15 systems. The system may be useful in automatically identifying and notifying proper 
law enforcement officials if a crime is being committed and/or a particular behavior 
being monitored is identified. The system may be useful for monitoring of parking 
security or moving traffic at intersections so as to automatically identify and track 
vehicle activity. The system may be configured to automatically determine if a 

20 vehicle is speeding or has performed some other traffic violation. Further, the system 
may be configured to automatically identify and characterize human behavior 
involving guns or human activity related to robberies or thefts. Similarly, the 
invention may be capable of identifying and understanding subtle behaviors involving 
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portions of body such as forelimb and can be applied to identify and understand 
human gesture recognition. This could help deaf individuals communicate. The 
invention may also be the basis for computer understanding of human gesture to 
enhance the present human-computer interface experience, where gestures will be 
5 used to interface with computers. The economic potential of applications in 
computer-human interface applications and in surveillance and monitoring 
applications is enormous. 

In one preferred embodiment illustrated in Figure 1, the invention includes a 
system in which an analog video camera 105 and a video storage/retrieval unit 110 

10 may be coupled to each other and to a video digitization/compression unit 115. The 
video camera 105 may provide a real time video image containing an object to be 
identified. The video storage/retrieval unit 110 may be, for example, a VCR, DVD, 
CD or hard disk unit. The video digitization/compression unit 115 is coupled to a 
computer 150 that is configured to automatically monitor a video image to identify, 

15 track and classify the actions (or state) of the object and its movements (or stillness) 
over time within a sequence of images. The digitization/compression unit 115 may 
convert analog video and audio into, for example, MPEG format, Real Player format, 
etc. The computer may be, for example, a personal computer, using either a 
Windows platform or a Unix platform, or a Macintosh computer and compatible 

20 platform. In one variation the computer may include a number of components such as 
(1) a data memory 151, for example, a hard drive or other type of volatile or non- 
volatile memory; (2) a program memory 152, for example, RAM, ROM, EEPROM, 
etc. that may be volatile or non-volatile memory; (3) a processor 153, for example, a 
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microprocessor; and (4) a second processor to manage the computation intensive 
features of the system, for example, a math coprocessor 1 54. The computer may also 
include a video processor such as an MPEG encoder/decoder. Although the computer 
150 has been shown in Figure 1 to include two memories (data memory 151 and 
5 program memory 152) and two processors (processor 153 and math co-processor 
154), in one variation the computer may include only a single processor and single 
memory device or more then two processors and more than two memory devices. 
Further, the computer 150 may be equipped with user interface components such as a 
keyboard 155, electronic mouse 156, and display unit 157. 
1 0 In one variation, the system may be simplified by using all digital components 

such as a digital video camera and a digital video storage/retrieval unit 110, which 
may be one integral unit. In this case, the video digitization/compression unit 115 
may not be needed. 

The computer is loaded and configured with custom software program(s) (or 
15 equipped with firmware) using, for example, MATLAB or C/C++ programming 
language, so as to analyze the digitized video for object identification and 
segmentation, tracking, and/or behavior/activity characterization. This software may 
be stored in, for example, a program memory 152 or data memory that may include 
ROM, RAM, CD ROM and/or a hard drive, etc. In one variation of the invention the 
20 software (or firmware) includes a unique background subtraction method which is 
more simple, efficient, and accurate than those previously known which will be 
discussed in detail below. 
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Referring to Figure 2, a general method of operation for one embodiment of 
the invention will be described. In operation, in the video analysis mode the system 
may receive incoming video images at step 205, from the video camera 105 in real 
time, pre-recorded from the video storage/retrieval unit 1 10, and/or a memory integral 
5 to the computer 150. If the video is in analog format, then the information is 
converted from analog to digital format and may be compressed by the video 
digitization/compression unit 115. The digital video images are then provided to the 
computer 150 for various computational intensive processing to identify and segment 
a predetermined object from the image. In a preferred embodiment, the object to be 

10 identified and whose activities are to be characterized is a moving object, for example 
a mouse, which has some movement from frame to frame or scene to scene in the 
video images and is generally in the foreground of the video images. In any case, at 
step 210 the digital images may be processed to identify and segregate a desired 
(predetermined) object from the various frames of incoming video. This process may 

15 be achieved using, for example, background subtraction, mixture modeling, robust 
estimation, and/or other processes. 

Next, at step 215, various movements (or still shapes) of the desired object 
may then be tracked from one frame or scene to another frame or scene of video 
images. As will be discussed in more detail below, this tracking may be achieved by, 

20 for example, tracking the outline contour of the object from one frame or scene to 
another as it varies from shape to shape and/or location to location. Next, at step 220, 
the changes in the motion of the object, such as the shapes and locations of the object 
of interest may be identified and their features extracted, and the positions of various 
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feature points or segments of the object such as animal head, animal tail, animal hind 
body etc. may be identified and their features extracted. Then, at step 225, the states 
of the object, for example the shape, location, and feature points and segments 
information may be used to characterize the objects activity into one of a number of 
5 pre-defined behaviors or events. For example, if the object is an animal, some pre- 
defined behaviors may include walking, turning, sniffing, etc. The pre-defined 
behaviors may be stored in a database in the data memory 151. 

Types of behavior may also be characterized using, for example, approaches 
such as rule-based label analysis or token parsing procedure. From these methods, the 

10 system may be capable of characterizing the object behavior as new behavior and 
particular temporal rhythm. 

The previous embodiments are generally applicable to identifying, tracking, 
and characterizing the activities of a particular object of interest present in a video 
image, e.g., an animal, a human, a vehicle, etc. However, the invention is also 

1 5 particularly applicable to the study and analysis of animals used for testing new drugs 
and/or genetic mutations. As such, a number of variations of the invention related to 
determining changes in behavior of mice will be described in more detail below using 
examples of video images obtained. 

One variation of the present invention is designed particularly for the purpose 

20 of automatically determining the behavioral characteristics of a mouse. The need for 
sensitive detection of novel phenotypes of genetically manipulated or drug- 
administered mice demands automation of analyses. Behavioral phenotypes are often 
best detected when mice are unconstrained by experimenter manipulation. Thus, 
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automation of analysis of behavior in a home cage would be a preferred means of 
detecting phenotypes resulting from gene manipulations or drug administrations. 
Automation of analysis as provided by the present invention will allow quantification 
of all behaviors and may provide analysis of the mouse's behavior as they vary across 
5 the daily cycle of activity. Because gene defects causing developmental disorders in 
humans usually result in changes in the daily rhythm of behavior, analysis of 
organized patterns of behavior across the day may be effective in detecting 
phenotypes in transgenic and targeted mutant mice. The automated system of the 
present invention may also detect behaviors that do not normally occur and present 
10 the investigator with video clips of such behavior without the investigator having to 
view an entire day or long period of mouse activity to manually identify the desired 
behavior. 

The systematically developed definition of mouse behavior that is detectable 
by the automated analysis of the present invention makes precise and quantitative 
1 5 analysis of the entire mouse behavior repertoire possible for the first time. The various 
computer algorithms included in the invention for automating behavior analysis based 
on the behavior definitions ensure accurate and efficient identification of mouse 
behaviors. In addition, the digital video analysis techniques of the present invention 
improves analysis of behavior by leading to: (1) decreased variance due to non- 
20 disturbed observation of the animal; (2) increased experiment sensitivity due to the 
greater number of behaviors sampled over a much longer time span than ever before 
possible; and (3) the potential to be applied to all common normative behavior 
patterns, capability to assess subtle behavioral states, and detection of changes of 
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behavior patterns in addition to individual behaviors. Development activities have 
been complete to validate various scientific definition of mouse behaviors and to 
create novel digital video processing algorithms for mouse tracking and behavior 
recognition, which are embody in software and hardware system according to the 
5 present invention. 

Various lighting options for videotaping have been evaluated. Lighting at 
night as well as with night vision cameras was evaluated. It has been determined that 
good quality video was obtained with normal commercial video cameras using dim 
red light, a frequency that is not visible to rodents. Videos were taken in a standard 

10 laboratory environment using commercially available cameras 105, for example a 
Sony analog camera, to ensure that the computer algorithms developed would be 
applicable to the quality of video available in the average laboratory. The 
commercially available cameras with white lighting gave good results during the 
daytime and dim red lighting gave good results at night time. 

15 Referring again to Figure 2, the first step in the analysis of mouse behavior is 

an automated initialization step that involves analysis of video images to identify the 
location and outline of the mouse, as indicated by step 210. Second, the location and 
outline of the mouse are tracked over time, as indicated by step 215. Performing the 
initialization step periodically may be used to reset any propagation errors that appear 

20 during the tracking step. As the mouse is tracked over time, its features including 
shape are extracted, and used for training and classifying the posture and body parts 
of the mouse from frame to frame, as indicated by step 220. Using this posture and 
body part information and all related information about the orientation, shape, and 
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position of the mouse generated for each frame, the actual behavior is determined by 
their relationship over time, as indicated by step 225. 

5 I. Location and Outline Identification and Feature Extraction 

The first step in analyzing a video of an animal and to analyze the behavior of 
the animal is to locate and extract the animal. A pre-generated background of the 
video clip in question is first obtained and it is used to determine the foreground 
objects by taking the intensity difference and applying a threshold procedure to 

10 remove noise. This step may involve threshold procedures on both the intensity and 
the. size of region. An 8-connection labeling procedure may be performed to screen 
out disconnected small noisy regions and improve the region that corresponds to the 
mouse. In the labeling process, all pixels in a frame will be assigned a label as 
foreground pixel or background pixel based on the threshold. The foreground pixels 

15 are further cleaned up by removing smaller components and leaving only the largest 
component as the foreground object. Those foreground pixels that border a 
background pixel form the contour for the object. The outline or contour of this 
foreground object is thus determined. The centroid (or center of mass) of the 
foreground object is calculated and is used for representing the location of the object 

20 (e.g., mouse). 

The contour representation can be used as features of the foreground object, in 
addition to other features that include but not limited to: centroid, the principal 
orientation angle of the object, the area (number of pixels), the eccentricity 
(roundness), and the aspect ratio of object. 

30 
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II. Mouse tracking 

Ideal tracking of foreground objects in the image domain involves a matching 
operation to be performed that identifies corresponding points from one frame to the 
5 next. This process may become computationally too consuming or expensive to 
perform in an efficient manner. Thus, one approach is to use approximations to the 
ideal case that can be accomplished in a short amount of time. For example, tracking 
the foreground object may be achieved by merely tracking the outline contour from 
one frame to the next in the feature space (i.e., identified foreground object image). 

10 In one variation of the invention, tracking is performed in the feature space, 

which provides a close approximation to tracking in the image domain. The features 
include the centroid, principal orientation angle of the object, area (number of pixels), 
eccentricity (roundness), and the aspect ratio of object with lengths measured along 
the secondary and primary axes of the object. In this case, let S be the set of pixels in 

15 the foreground object, A denote the area in number of pixels, (C x ,C y ) denote the 
centroid, <f> denote the orientation angle, E denote the eccentricity, and R denote the 
aspect ratio. Then, 



20 Let us define three intermediate terms, called second order moments, 

!>0 =£(*-C,) 2 m oa =2>-C,) 2 mv =^(x-C x )(y-C y ) 



m 2,i 
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Using the central moments, we define, 
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<p = — arctan : 

2 m 20 - m 02 

E ^ (m 20 -m Q2 ) 2 +4m u 2 
(m 2Q +m 02 ) 2 

R is equal to the ratio of the length of the range of the points projected along an axis 
perpendicular to fa to the length of the range of the points projected along an axis 
5 parallel to <j>. This may also be defined as the aspect ratio (ratio of width to length) 
after rotating the foreground object by <f>. 

Tracking in the feature space involves following feature values from one 
frame to the next. For example, if the area steadily increases, it could mean that the 
mouse is coming out of a cuddled up position to a more elongated position, or that it 
10 could be moving from a front view to a side view, etc. If the position of the centroid 
of the mouse moves up, it means that the mouse may be rearing up on its hind legs. 
Similarly, if the angle of orientation changes from horizontal to vertical, it may be 
rearing up. These changes can be analyzed with combinations of features also. 

However, it is possible for a contour representation to be used to perform near- 
15 optimal tracking efficiently in the image domain (i.e., the complete image before 
background is subtracted). 

HI. Mouse feature points and segments identification 

Once the features are obtained for the frames in the video sequence, the 
20 foreground state of the mouse is determined by identifying certain important feature 
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points and segments on the mouse such as the head, tail, waist, fore body and hind 
body. 

The head is detected by using a combination of features including direction of 
motion, distance to tip from the center of mass, the curvature at that point, and the tail 
5 information. The head is not on the side of the tail, but, on the other side. 

The tail is detected using a combination of shape features including thickness 
information and distance from center of mass. 

The waist is detected by determining the minor axis of the ovoid shape of the 
animal after having the tail removed from consideration. 
1 0 The portion of the body in front of the waist towards the head is called the fore 

body and the portion of the body aft of the waist towards the tail is called the hind 
body. 

Using these identified points and segments, various parameters such as the 
orientation, heading direction, turning angle, proximity to other objects or zone 
15 boundaries, etc. are obtained. 

IV. Behavior Detection Methodology 

Each behavior can be modeled as a set of rules or conditions that must be 
20 satisfied. The rules or conditions can be formulated using any of the available 
features or parameters including position and shape of specific body parts with or 
without respect to other objects, motion characteristics of the entire mouse body or 
individual body parts, etc. In the descriptions below, all such rules or conditions that 
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are required to derive the specific modeling of the behavior are stated. The behavior 
descriptions follow: 

A. Freeze 

5 Freezing behavior is determined by the absence of movement of rodent body for a 
brief period of time. Freezing behavior is primarily used in conditioning-fear 
experiments where auditory tones or electric shock is administered to the animal to 
cause fear leading to freezing. 

10 B. Sniff at Objects 

Sniffing behavior is determined by the touching of the mouth of the rodent body 
against another defined object, or animal. A zone or area is calibrated to represent 
this target object or animal, and any encroachment or contact of the head/mouth of the 
rodent body against or into this target zone is detected as a sniff. 

15 

C. Locomote 

Locomotion behavior is defined as the movement behavior of the rodent around the 
cage or arena. Locomotion behavior is best viewed from the top thereby allowing 
accurate measurements of the total distance traveled, speed, acceleration, heading 
20 direction, turning angle, and distance to a specific target. 

D. Stretch and Attend 

Stretch and Attend behavior is determined by the purposeful extension of the head 
portion of the rodent body forward and a subsequent retraction of the head while the 
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hind part of the rodent body remains stationary, when viewed from the top. The 
extension of the head may involve bending it to the side. 

E. Head Dip 

5 Head Dipping behavior is determined by the downward movement of the head of the 
animal over a ledge or a platform, as if to look below. This can be either intentional 
or unintentional. 

F. Transgress from one area to another 

1 0 Transgression behavior is detected by the movement of a portion of, or the entire 

body of the rodent across from one defined zone or area into another defined zone or 
area. With this behavior, exit and entrance measures with respect to a zone can be 
calculated. 

15 V. Behavior identification 

Using the feature data assigned for each of the frames in the video clip, the 
approach is to determine those behaviors and events as defined in the previous step. 
This process will be accomplished in real-time so that immediate results will be 
reported to investigators or stored in a database. One approach is to use a rule-based 

20 analysis procedure by which various features are analyzed and if these features fit 
certain criteria then, that particular behavior or event is detected. For example, "when 
the mouth touches a object in an object recognition paradigm and remains in contact 
for a minimum duration of time and then, releases contact with that object and 
remains out of contact for a certain minimum duration of time, that episode is called 

25 "Sniffing". 
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In summary, when a new video clip is analyzed, the system of the present 
invention first obtains the video image background and uses it to identify the 
foreground objects. Then, features are extracted from the foreground objects and 
various important feature points and segments on the animal are identified. This set 
5 of features is passed to a behavior identification system module that identifies the 
final set of behaviors or events for the video clip. The image resolution of the system 
that has been obtained and the accuracy of identification of the behaviors attempted so 
far have been very good and resulted in an effective automated video image object 
recognition and behavior characterization system. 

10 The invention may identify some abnormal behavior by using video image 

information (for example, stored in memory) of known abnormal animals to build a 
video profile for that behavior. Further, abnormalities may also result from an 
increase in any particular type of normal behavior. Detection of such new abnormal 
behaviors may be achieved by the present invention detecting, for example, segments 

15 of behavior that do not fit the standard profile. The standard profile may be 
developed for a particular strain of mouse whereas detection of abnormal amounts of 
a normal behavior can be detected by comparison to the statistical properties of the 
standard profile. Thus, the automated analysis of the present invention may be used 
to build a profile of the behaviors, their amount, duration, and daily cycle for each 

20 animal, for example each commonly used strain of mice. A plurality of such profiles 
may be stored in, for example, a database in a data memory of the computer. One or 
more of these profiles may then be compared to a mouse in question and difference 
from the profile expressed quantitatively. 
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The techniques developed with the present invention for automation of the 
categorization and quantification of all mouse behaviors is a powerful tool for 
detecting phenotypic effects of gene manipulations in mice. As previously discussed, 
this technology is extendable to other behavior studies of animals and humans, as well 
5 as surveillance purposes. In any case, the present invention has proven to be a 
significant achievement in creating an automated system and methods for automated 
accurate identification, tracking and behavior categorization of an object whose image 
is captured in a video image. 

In another preferred embodiment of the invention, there are multiple cameras 

10 taking video images of experiment apparatus that contain animals. There is at least 
one apparatus, but, as many as the computer computing power allows, say four (4) or 
sixteen (16) or even more, can be analyzed! See Figure 5. Each apparatus 510 
contains at least one animal or multiple animals. The single or multiple cameras 505 
may be taking video from different points of views such as one taking video images 

15 from the side of the apparatus, or one taking video images from the top of the 
apparatus. These apparatus can be home cage, open field cage, water maze device, T- 
maze device, Y-maze device, radial arm maze device, zero maze device, elevated plus 
maze device, or other experiment devices. The invention can also be applied to 
various experimental paradigms such as object recognition, and conditioned fear 

20 freezing experiments. When video images are taken of multiple apparatuses and 
devices containing one or multiple animals, and are analyzed for identifying these 
animals 5 behaviors, high throughput screening is achieved. When video images taken 
from different points of views, for example, one from the top view and another from 

37 



PATENT APPLICATION 

Attorney Docket No.258/160 

the side view, are combined to identify animal's behaviors, integrated analysis is 
achieved. 

A variation of the present invention is directed to automatically determining 
locomotion behavior of mice or rats in an open field 311. Once the animal is 
5 identified as a foreground object 308 as discussed above, the body parts of the animal 
such as head and tail, and hind limbs and forelimbs, and center of mass, are identified 
310. The traces of the path of the movements of the animal's center of mass in the 
open field under observation is recorded, its instant and average speed of movements 
and distance traveled are calculated, its instant and cumulative body turning angles 

10 are analyzed 311. In addition, events like turning ratio (ratio of path length over 
number of turns, where number of turns is counted when the animal makes a turn 
larger than 80 degrees when the animal travels one body length); proximity score 
(calculated by determining the distance of the animal from the goal during each 
second of the trial and is used as a measure of deviation from the ideal path to the 

15 platform once an animal is placed in the cage); heading errors (defined as an instance 
of moving away from a target); and animal staying in a specific zone inside the field, 
are recorded 316. Then a visualization process will further analyze the result of the 
path trace recorded to generate variety of statistic results. Visualization process 
allows users to use graphic drawing tools to define any number of zones of any shape 

20 in the open field as needed. The system provides a graphic tool that allows users to 
define the field of any shape. An example apparatus is shown in Figure 5, where a 
camera 505, placed directly above the arenas 510, captures video from the top. A 
plurality of the arenas may be used and their video captured using a single camera or 
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multiple cameras to achieve high-throughput analysis. The locomotion or movement 
behavior of the animal in each arena is analyzed. 

Another variation of the present invention is directed to automatically 
determining the object recognition activity 312. Graphic tools are provided to allow 
5 users to define objects in the scene. Once the animal is identified as a foreground 
object 308 as discussed above, the body parts of the animal such as head and tail and 
hind limbs and forelimbs, and center of mass are identified 310. The traces of the 
path of the movements of the animal's center of mass are recorded. The distances of 
the animal's head to any of the objects in the scene are calculated and when the 

10 distance to an object is less than a user-defined amount, the animal is counted as 
animal's sniffing at the object and is said to be exploring that object 312. Statistics 
about these exploring events are generated and exported 317. An example apparatus 
is shown in Figure 7, where a camera 705, placed directly above the arena 710, 
capture video from the top. A plurality of objects, in this case two objects 715 720 are 

15 placed in the arena and the exploratory or sniffing behaviors of the animal on these 
objects are analyzed. 

Another variation of the previous embodiments is particularly applicable to the 
study and analysis of mice or rats in their spatial learning and memory. This third 
variation of the present invention is directed towards automatically determining the 

20 behaviors of mice or rats in a water maze experiment environment 3 1 5. Graphic tools 
are provided to allow users to define the maze and platforms. Once the animal is 
identified as a foreground object 308 as discussed above, the body parts of the animal 
such as head and tail and hind limbs and forelimbs, and center of mass are identified 
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310. The traces of the path of the movements of the animal's center of mass are 
recorded. The latency (the time period the animal spent in swimming in the water 
before landing at the platform) is measured; its instant and average speed of 
movements and distance traveled are calculated; its instant and cumulative body 
5 turning angles are analyzed 315. In addition, events like turning ratio (ratio of path 
length over number of turns, where number of turns is counted when the animal 
makes a turn larger than 90 degrees when the animal travels one body length); 
proximity score (calculated by determining the distance of the animal from the goal 
(platform) during each second of the trial and is used as a measure of deviation from 

10 the ideal path to the platform once an animal is placed in the water); heading errors 
(defined as an instance of swimming away from the VISIBLE platform); and animal 
staying in a specific zone inside the maze, are recorded 320. Then a visualization 
process will further analyze the result of the path trace recorded to generate variety of 
statistic results. Visualization process allows users to use graphic drawing tools to 

15 define any number of zones of any shape in the open field as needed. The system 
provides a graphic tool that allows users to define the field of any shape. An example 
apparatus is shown in Figure 8, where a camera 805, placed directly above the water 
tank 810, captures video from the top and the movement behavior of the animal inside 
the water tank and its relationship with the target platform 81 5 is analyzed. 

20 Another variation of the previous embodiments is particularly applicable to the 

study and analysis of mice or rats in their spatial learning and memory and anxiety. 
This fourth variation of the present invention is directed towards automatically 
determining the behaviors of mice or rats in a variety of maze apparatus 313. Graphic 
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tools are provided to define specific maze apparatus, such as T-maze, Y-maze, radial 
arm maze, zero maze, elevated plus maze, and etc. Once the animal is identified as a 
foreground object 308 as discussed above, the body parts of the animal such as head 
and tail and hind limbs and forelimbs, and center of mass are identified 310. The 
5 traces of the path of the movements of the animal's center of mass are recorded. 
More importantly, the animal's behaviors related to every arm in the maze, such as 
time spent in each arm, the number of times entering and exiting an arm, are found 

313. Besides, animal's instant and average speed of movements and distance traveled 
are calculated; its instant and cumulative body turning angles are analyzed 318. In 

10 addition, events such as animal partial incursions into particular arm (for example, the 
animal might maintain its hind quarters in a closed arm while poking its nose into an 
open arm); Stretch-Attend Behavior; Head-Dipping behavior; and Supported Rearing, 
are detected 313. An example apparatus is shown in Figure 6 where an elevated plus 
maze 610 is used to analyze the behavior of the animal. A camera 605, placed directly 

1 5 above, capture video from the top and analyzes the movement behaviors in various 
arms or areas of the maze. 

Fifth variation of the present invention is directed to automatically 
determining the freezing behaviors of mice or rats in a cued or conditioned fear tests 

314. Graphic tools are provided. Graphic tools are provided to define the area within 
20 which animal activity is measured. Differences between neighboring frames are 

compared pixel-by-pixel in terms of their intensity and color intensity. These 
differences are used to calculate the motion of the animal from frame-to-frame 
because motion in the area is caused by movements of the animal. The values of 
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these differences indicate if the animal is moving or freezing 314. An example 
apparatus is shown is Figure 9. The camera 905 maybe placed inside or outside the [ 
chamber 910. The animal is placed inside the chamber and stimuli in the form of ! 
auditory tones 915 or electric shock 920 is presented and their behavioral effects 
5 following the stimuli are analyzed. 

The systematically developed definitions of mouse behaviors that are , 
detectable by the automated analysis according to the present invention makes precise 
and quantitative analysis of the entire mouse behavior repertoire possible for the first 
time. The various computer algorithms included in the invention for automating 

10 behavior analysis based on the behavior definitions ensure accurate and efficient 
identification of mouse behaviors. In addition, the digital video analysis techniques of 
the present invention improves analysis of behavior by leading to: (1) decreased 
variance due to non-disturbed observation of the animal; (2) increased experiment 
sensitivity due to the greater number of behaviors sampled over a much longer time 

15 span than ever before possible; and (3) the potential to be applied to all common 
normative behavior patterns, capability to assess subtle behavioral states, and 
detection of changes of behavior patterns in addition to individual behaviors. 

Although particular embodiments of the present invention have been shown 
and described, it will be understood that it is not intended to limit the invention to the 

20 preferred or disclosed embodiments, and it will be obvious to those skilled in the art 
that various changes and modifications may be made without departing from the spirit 
and scope of the present invention. Thus, the invention is intended to cover 
alternatives, modifications, and equivalents, which may be included within the spirit 

42 
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and scope of the invention as defined by the claims. 

For example, the present invention may also include audio analysis and/or 
multiple camera analysis. The video image analysis may be augmented with audio 
analysis since audio is typically included with most video systems today. As such, 
5 audio may be an additional variable used to determine and classify a particular objects 
behavior. Further, in another variation, the analysis may be expanded to video image 
analysis of multiple objects, for example mice, and their social interaction with one 
another. In a still further variation, the system may include multiple cameras 
providing one or more planes of view of an object to be analyzed. In an even further 

10 variation, the camera may be located in remote locations and the video images sent 
via the Internet for analysis by a server at another site. In fact, the standard object 
behavior data and/or database may be housed in a remote location and the data files 
may be downloaded to a stand alone analysis system via the Internet, in accordance 
with the present invention. These additional features/functions add versatility to the 

15 present invention and may improve the behavior characterization capabilities of the 
present invention to thereby achieve object behavior categorization which is nearly 
perfect to that of a human observer for a broad spectrum of applications. 

All publications, patents, and patent applications cited herein are hereby 
incorporated by reference in their entirety for all purposes. 

20 



43 



